Recently, there has been a trend toward high integration of a semiconductor memory device and thereby a stacked semiconductor memory device has been proposed. In a stacked semiconductor memory device, a stacked body in which word lines and interlayer insulating members are stacked alternately and a memory hole that penetrates through the stacked body are formed and a memory member is provided on a side surface of the memory hole. The memory member is formed to have a block insulating member, a charge storage member, and a tunnel insulating member, which are stacked from the outer side in this order. A silicon pillar and an insulation member are provided further on the center axis side from the memory member.